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ABSTRACT 

v * On May Iff, 1583 a conference was held at the 
, Northwest .Regional Educational Laboratory ( NWREL ) to discuss the 

status" of Hork on cost analysis studies in education at the state and 
local level, NWREL, local and state education agencies in Oregon and 
Washington , and two Portland area universities were represented- at 
the conference Jana "Kay Smith presented a report which reviewed cost 
studies conducted- at NWREL from 1978 to 1982, The report assessed the 
current practice of cost analysis * in education and proposed; a system 
to help determine theldbst iieeds of ^ client requesting a cost study, 
\ Lapry Picus had conducted the majority of co^t studies reviewed in 
Smith 1 s report and shared Vhis experiences with applications of cost 
analysis methods in educational setting*?. Members of a panel which 
^consisted of loCal,and state education agency and NWREL staff 
discussed prospects for cost analysis methods in their agencies* All 
panel members (Don Egge / Walt Hathaway , Zeno Katterle, and Robert 
Rath) "were known for their interest in the area of cost analysis and 
for their experience in /the conduct pf such studies, A transcript of 
the cpxitetencfe proceedings (major presentations and audience 
d^isfiussion ) is included. (PN) 1 " v . .... 
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PREFACE^ 



The Research on Evaluation Program is a Northw^t Regional 
Educational Laboratory project of research, development 9 tea 
and training designed to create new evaluation methodologies 
use in education^) Tn i s document is one of a series of paper 
reports produqep by program staff # visiting scholars* ad junc 
scholars, and "project collaborators — all members o£ a cooper 
network of colleagues working on the development of new 
methodologies* ' . ■ * • 
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for 
s ^and 
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ative 




What cost analysis work is currently being done in school : ad 
state departments? Is cost-effectiveness analysis a truly 
pragmatic alternative method for doing educational program 
evaluations? These and related questions are addressed in this 
report which contains a transcript of a conference held to 
discuss the practice, problems, and potential of doing 
educational cost studies. Representatives of a regional 
laboratory, a state department of education/ and two sphool 
districts recount their own 'experiences and provide unique 
perspectives on the potential utility of educational cost 
analysis ' studies . 



Nick L. Smith,, Editor 
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</:\ On May 18, 1083 a conference was held at nwrel to discuss the 
status, of work on cost analysis .studies, in education at the state, 
;and local; level. The conference was\ based ori a report completed - 
recently by J ana Kay Smith entitled "Case Reports of Northwest 
Regional Educational Laboratory Cost* studies" which examined 16 \ 
educational goat studies conducted by NWREL staff. Earlier work * 
by Henry 'ii- Levin had identified four predominant methods of cost 
analysis in educational evaluations /cost-benefit* . .oo4t-affeotiwp-\ 
ness, cost-utility, and cost^fi^asibility • Because Smith had not ' 
found applications of these methods , one reason for the confer- 
ence was tp try to learn if these four methods of cost analysis. : ; 
were being employed By state, and ulocal education evalua tors* The 
conference also presented ah opportunity to (a) review smith's , 
report, (b) discuss the problems of^ doing cost, studies, and 
(e) explore the future of cost analyses in educational evaluation. 

Twenty- two people attended the Conference, including 
representatives from NWREIu local and state education agencies in 
Oregon and Washing*ton7"a£ well as two Portland area universities. 
The following presenters were invited |to, participate; , ^ 

The first featured speaker was Jana Kay Smith * a visiting 
scholar with the Research on Evaluation Program. Jana presented 
her recently completed report ^which reviewed cost studies { 
conducted at the Northwest Regional Educational laboratory 
(NWREL) " from 1978 to 1982# The* report assessed the current ■*■-" 
practice of cost analysis in education and, proposed a system to 
help determine the cost neetis of -a client requesting a cost study* 

The - second featured speaker! was Larry Pious * an economist 
with the Center for state Studies of NWREk- Larry had conducted' 
the majority of cost studies reviewed in Smith's report and 
shared' His experiences with^ applications of cost analysis methods 
in educational settings* /\ .• ?■ 
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' ' ' - ' ■ ' 1 * 

Finally, member s of a panel which consisted of local and 
state education agency and .NWRED staff discussed prospects for * 
coat analysis methods, in their agencies. % All panel- members were 
known for their interest in the area of, cost analysis and for * 
their experience in toe conduct of. such studies, panel members 
included: Don ' Igge , Oregon State Department of Education f Walt 
Hathaway ,.. Portland Public. Schools; Zeno' Katter ie j Beaverton 
School District! and Robert Rath , NWREL. * : " 

Thm rest of this document contains a transcript of the * 
conference proceedings, including major presentations and 
audience discussion. Its purpose is to (a) make available the 
ideas shared by the presenters , (b) stimulate debate on the 
feasibility of conducting cost analyses in educational 
evaluation, and (c) provide a practical perspective ori past, 
present 9 and future^ application of cost analyses at local and 
state levels, * 
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= * ; CONFERENCE PROCEEDINGS *. . 

„..•'/ V ■ ;•/ .. •'• . •: , ' / 

Jana Kay Smith , ; . ■ * , ■ 

i have „ already shared with you copies of my report entitled, 
Case Reports of Northwest Regional jMucational Laboratory Cost _= 
Studies, The purpose of the study ^escribed in it ( was to 
document the state of practice of cost studies in educational * 
settings. The /way that we did this was, first ofi all, to 
identify projects that had taken place i second, to abstract from 
the written reports the procedures that were used in the 
•projects^ and finally, to conclude with interviews with the 
principal investigators to get subjective information on investi- 
gator and client satisfaction* information on utilization, , and so 
on. *' 

I identified 15 cost studies that fell into four categories. 
If you refer to Table 1 in your reports, you 1 11 see that within 
* category one I found five projects (See Table 1) , These projects 
consisted of cost comparisons between two qr more programs or 
entities- In category two there were three projects that 
described the costs of a t single program or entity. In category 
three, budgets and planning, there were three studies, all of 
which looked at budgeting or planning issues, m And, finally, 
there were four reports in the category I called policy analysis,' 

I was quite surprised, when 1^ tried to categorize these ■ 
reports* \ I expected that the reports would fall into Levin's 
four cost categories i eost^eff eetiveness, cost-benefit, cost- 
feasibility, and cost-utility. /However* when I tried to fit *tho 
Reports into those those categories, I couldn't. So I went back 
to the studies for a second look at them to see what factors had 
led to the types of methods that were used, and whether or not 
alternative cost-analysis methods could have been possible - 

During the second review, I did find some cases where 

* ■. 

alternative cost analyses were possible. For example, 1^ didn't, 
classify Cases 1, 2, and 3 as cost-feasibility analyses, although, 
in some sense they could be interpreted as such. Cost ' : 

i ' . ■ . ■ ' 
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LEA s Lec^l Bducation Agency 
SEA * State BdUGition Afency= = 
. CSSO = Chief State .School flf fibers 
L® -t AVailible i 



feasibility analysis tabulates the cpsts for a" given program or 
programs and compares those, costs to a ceiling^ limit. In 'other 
words 9 is it feasible within our economic structure toT support 
th is prog ram? Cases 1* 2 g and 3 >wer e not eons idered to be t 
cost-feasibility "methods, because there was no explicits rnent^Dn- 
of upper limits, or points beyond^ whicii an alternative" was no A * 
longer economically feasible. * \ f ; 

^Another ^lter native ^analysis that * I- thought was possible for. 
these three studies was cost-effectiveness. "\ Cost-effectiveness - 
analysis forms a ratio between the cost of the program and a 

non-monetary effectiveness measure, such as test scopes. I 

* ■ ■ • • • . • - * . 

didn f t consider Cases *l r 2, and 3 to be cost-effectiveness 
studies, because there * was no ; mention .of the effectiveness of the 
Alternative's , although it was/assumed by the principal invest i- . 
gator that each alternative provided the 'same quality of 
transportation . In' this sense, effectiveness was assumed to be 
eqteal across all alternatives and was . not taken explicitly into 
account. 5 It could 'be argued, therefore, that the effectiveness 
component off the bost^ef f ectiveneas . ratio Was equivalent and so 
dropped out* leaving the cost figures to stand alone. 

At first/ I thought that such an assumption was inappropriate. 
Since then, I've been doing more research in the; literature , and 
repeatedly found instances of the same assumption. One . report 
that Nick /Smith and 1 ran across reviewed ^14 studies assessing 
the cost-effectiveness of home health care versus hosprtal health 
care * All of" the rev iewed stud ies assumed* that the alternative 
care programs achieved the same outcome . Although the title .of 
those articles implied a cost-eff^ectiveness analysis of home 
fiealth care programs, when you looked at the "stuejles , there were 
ho measures of effectiveness. Effectiveness -was" assumed to be 
equivalent across programs and was "dropped out of thefcost- 
effectiveness ratio* - - ! , ' ■ 'i- ; * 

- T In Case 4, the Hawaii Secondary Analysis^ cost was a minor 1 " 

but' interesting part of the study. This ! etucty wa~s intended to 

• ■- : - - * . '■ ' . ♦ * * - " r ! _" ■ - =\ • •• = 

provide a prescriptive model for utilization^ of -existing data. 

What I found in the report was that cost data were readily^ 



available^ werVvery available, and yet 

^ Jihere was no_at£@rapt to Jput jAese two together • When I talked 1 
j about Wis witii the principal investigator* however, a rationale 
-for not con^ning'tha effectiveness and cost data^was provided.- * 
• v There *was, no way to insure that the programs wet£ uniformly * 
'; r [ V. : implemented across schools i so it %ld;have been inappropr lata 

to pool the data and form Gost^ffe^tiveness ratios, v \ -. 

-y- in Wis case f ; it was correct not to use cost-effectiveness 
analysis , However f if the report was prescriptive in order to ;-. 
provide a, model for evaluation, why not encourage an attempt to 
insure tmif orroity of impiemeritafei^ you " . / /• 

. " cpu Id. then combine the cost Mid effectiveness data iijto a ratio? ^ ^ 
I also felt-that in Cases 6 and 7 (although I may not be y. • V 
■17 entirely correct in this) there could have been a potential for a -•' : 
cost-effectiveness ari&Iy sis » These were federally implemented =, ; : 
'programs, where I felt that there were most probably some 
. "measures of effectiveness available which.. could _haye>;been ; : 
compared to the program costs. a !\ " 

Finally, Case 8 refers to Adopter 's Guides' produced by the 
Lab j^hich are designed to provide three perspectives on 
implementation of a program: (1) how to; develop a program, 

(2) how to determine the costs of developing and running it, and \ 

(3) how to measure its effectiveness. In the Adopter f s Guides - : 
that I looked at, there was. no mention, even though there were 

■Z models provided for evaluation , •• of the pos s ib i 1 ity o f combining ^ 

the cost data with the effectiveness data* This seemed to be an 
/ emission that could be easily corrected. ■-■ In these situations, /% 

• given that the guides were prescriptive, I felt cos t-e f f ec t i ve- 
'>'•• Vness methods could have been suggested* , . . y ' y''^ ? ;.", 

V *"".* The question I have is , "Why, when, the data were -available in \-=\- 
some cases r and readily available in others, weren't these more ^ 
soph isticm ted cost analyses done?" " Was there not an interest in 
the reiatic«ship of cost to effects?* If you will refer to * ; 
4 • Figure iVvI developed a flow chart in .order Jto understand better 
why particular cost methods were chosen , , 
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Figure li Factors affecting" the choice of a cost analysis method* 
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First, I wondered why people don't ask about cost^ff ective- , 
nesi, or ask questions o£^a more sophisticated natura, :It may be 
that the clients 4on!t know that they *3an ask these questioner, or 
they do>n?t know ;ta_.s§y /jfigt j&a^^n^ ito : see how 1 cost relates to 
effects* ; /ihat could bCVOM ^^sony ; v '•: ; V t 

: Second , it could bet: tha tether e ;is ho reason to relate cost to 
effects. In. that case, c^t^ffectiveness or cost-benefit 
'^^•i^l^ses would be inappropriate: you would be giving more to 
^S^r • plien^ or than they would use, ^ > 

• : A third the one -that is of most interest to 

me, is : whethe£^r^not it ! s feasible to do a cost-effectiveness 
study V In the cases I reviewed, it seemed that it was very 
feasible ; the_ data were already avail able! However, I have 
frequently heard, arid M may hear about this from our panel • 
members, that usually it f s simply too difficult to get the data . 
you need for a eos toffee tiyerifss_ study - . Comparative programs . 
are not in existence. . You .can 1 1 . get uniform measures abross _ _ Z 
programs, and so on* It will be interesting to hear other 
people's perspectives on that, I won't run through Figure;! 
because I think it's self-explanatory. Basically, in the figure 
what I do is. follow through the various alternatives I just 
mentioned and suggest the appropriate role for the evaluator, 
whether it be to carry out the request of the client, to educate 
"the client on cost-analysis methods, or to suggest alternative' ..- 
methods ... : * ' . .\> ' V 

Finally, in the report, I looked at implications for future 
cost studies. People seem not to know about coist-eff ectiveness 
analysis | perhaps we need to educate clients, about the informa- 
tion they can learn from their data about their programs. 
Conversely, it may be that there is actually not that much 
interest in, or need to, relate cost to 'effect. In this case, we 
should, be developing methods that measure cost-feasibility or 
budgeting. Finally, it may be that cost-effectiveness analyses 
are Simply not "doable' 1 or feasible for educational evaluation. 
Again, in this case, perhaps we should be promoting cost methods 
that are more reasonable or "doable,^ : =/* : 
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#ffTCt4yin#ss^^tft-banefit, cost-feasibility, eoet-utJ^ty> ancl 
has promote^! them as being the primary methods oi oost'an^lysi^,: 
That is not what I found; Perhaps we need to investigate other* 
c6s^a«aiysis methods* Hopefully # ; the peispect^^^ 
mexpb^r s w ill - he lp d iar if y th is issue:* * and v wllly project ;fojr us: the}: 
neCT fob the different types of methods. In add it ion r it wi 11 fee 
^Interesting to hear_afrout the successes that they ? ve had _ with , 
these, methods, and techniques that they 've used in the past," 

Aud i ence Discussion -}--. # V\;.-v^ ■ :< ^ i "'\ •••:-"''. '■:}'■- / v---, 

guestion r Will" you you run through for me again the defini- 
tXc^Th^^ 

cost-feasibility? These are not well def ined . in the literature , .-, 

Answer ; _ Cost-feasibility analysis tabulates the cost of the 
program? it takes into account cos^s alone and compares those to 
a ceiling cost- For example ,. suppose you have $10 f 000 to spend 
on a program. It will cost $12,000 to implement program A and 
$9,000 to implement Program B; it is not feasible to implement 
Program A. According to our. feasibility standard, we have to 
implement Program B. That's cost-feasibility* 

Cost-utility uses a combination of subjective weights and 
expected outcomes* you don't take any measures of outcome , but 
rather estimate outcomes and place importance weights on those 
probable outcomes. It \b the same logic as cost-effectiveness, 
except that you ?re guessing what the .outcomes would be pi stead; of 
measuring them- . _ • ' .. . \ . ^ 

Cost-effectiveness forms a ratio between a non -monetary 
measure of effectiveness, such as a. test score and the program 
cost, Any traditional outcome data could be used as effective- 
ness, data, = ■"'•-' ..■ ■ - 

Cost-benefit places a monetary vklue on the outcome, - For 
example, computer training may result in a benefit of $5,000 
increased yearly income. You form a ratio between that monetary 
value and the cost of the program. Does that clarify it a bit? 



ERIC 



Answer: Yes* 



Laj?ry. 



V As ,there-;are no ; other questions / t will give the floor to / 



. ■ * 



totry Pious 

Thank you, I was asked /to speak at this colloquium largely 
because I was the ;principali investigator |f or the -vas^t majority of • 
the studies teat were cr itigued In this report* I spent a lot of 

is^ery^njpy_able._ 



time talking about them with J ana Kay, r 

It is nice to have 'someone /at the Lab interested in' the work I've 

:^M.J J..... «... '•- ..'; = ":.. /••...« / .■ ; , " ; 



report, I was struck by tne disparity between the four ca tegor ies 
that Levin suggests as tne way to approach cost analysis , and the 
four categories that encompassed the studies we've done* I'm ^ 



wofcder ing why th is discrepancy occurred . 

t f ve been familiar jkith' Levin's work and his articles for 
most of the time I've seen at the Lab. Why is it that we 'ye gone 
off in a very different direction? /There are probably two 
reasons » ' One has" to do with the nature of the client requests. 
We * were usually asked / to provide the client with a description of 
, program costs, so that is what we did. For example, in the first 



two studies. Numbers / l and la, whi'ch were transportation studies, 
we were asked to compare the costs of two alternative transporta- 
tion systems. It was assumed that the outcomes of either option 
were Identical. A certain number of children would be trans- ^ 
por ted ever y day to certa in locations ; and we 'We r en ' t asked to 
consider differences in the quality of the transportation 
alternatives. / 

In other studies, such as the student activities study that 
we did with a number , of districts in southwest Washington, 
measures of outcomes seemed very difficult to obtain. We we re 
looking at extracurricular activities, such as bands, fooc^ull* 
teams, etc. , where outcome data were difficult to obtain and y 
measure. Even trying to come up with some cast information was 
yery, very difficult. In my view, the major goal of that study 
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was to try 'to help the, school d iptr io ts establish a framework so j 

" - ' - ' - \ . \ -- ' -■ ' \ _-' ' '"- f 

that " they could conduct post comparisons between programs in^the^ ; 

futuie.v. Similarly, in . Cases 6; and 7 -wt were tasked :^::d4scri^ : .>l^ 

programs for, special student populations, and for Southeast Asian 

students* We wer en 1 1 look ing at outcomes because we jwer en it r fu 

specifically asked to do so- \ ' - ' , , f , , j 

The program under :^ich many of the studies were" done * has ;; ;);• . 

primarily tried to be responsive to /client requests* i found, ; / ^= 

Jaha Kay's flow chart to be Very helpful; it suggests that :^ i : 

perhaps it's the r esppns ib illty of the contractor , the eyaiuator >./ 

or economis ^suggest to the client th^t rather methods are • / r 

. - • *- .- , 1 ■. y ■ . j ' ' ■■ '■■ . _ - • t,\ .• ■ . - . : . . y =".:.* j 

available* .Hopef ally; ^w^ 

future work, " ' / .": ".. ". - : : - ; ; > :'*.. 

The second reason that/ the categor ies in our work "were 
different from Levin f s r in/ the "simplest form, is that I f m not an 
evaluator, I hadn't thought of these studies as evaluations, but 
rather ^s - cost studies or'cdst descriptions. While Ve valuators ^ - 
often don't think of comparing outcomes with costs^ as ail - 
economist, perhaps I didn * t think of ten enough about comparing 
the costs to outcomes. Both sides need to be considered and 
brought together - " 

What I f d like to do now, is talk about w^ys that school 
districts and; the tobo^atory can conduct cost analysis studies in 
the future • /We need to look at both the short term and the long 
term. In the v long term,; cost-effectiveness and th<^ cost-benefit 
analyses require a? great deal of ^effort and time, In particular, . 



cost-^eff ectiveness, which Levin to feel is the most appro- 

pr iate in - .education , requires good evaluation measure s . 1 When you 
start a program, you don * t have any outcome data available 
immediately. For example, in the early days of Title I, the 
expected improvement in student achievement took m^eh longer to 
show up than anybody expected *~ People sort of thought that with 
all of the money being spent on the program, the kids would do 
better immediately* However, if I understand the literature ? 
correctly, it took nearly "five year's for these gains to begin to 
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V. show up. Cost-effectiveness analysis has a positive f utuce if ' : 
plans ar e made now to N cd Hep t data , for future use!. > .. 



In 3 recent iss ue Education Evaluation land Policy 
ysis , Stanley Pog row su^aajgfg that need to move b 



at the cost^ide^jof; things, and hold the Effectiveness •measures 
5as a constant, ^ ; ttotigh t that tn%t was % little' extreme > as i'mj 
not sure we should sa^i ^Here's what our outcome is i how can we 
do it "Jot leas money? m We v certainly need to look at ways ip 
[whi^:^ can increase productivity as we 11 asv dl'ecr ease costs in 
;the schools .- In that easf ~, X^^i^^^^^^^^^^ii^^^ - r 
proposes, particularly cost^ 

Over a period of years * we can^evaXuate results of alternative T _ 
approaches to some p r ogr ams * and w# can say^pDid we the same 
tiling withriess i money^ or did it test ufeifcc = |^?i^f ^Hti^can we 

In the short terra, school -districts 

3ced 



direct re sources to max imize our outcomes 11 /^u^ ^ ^ - 

still : ^ve-to^^ke 
management- and budget: decisions . • Every schooled istrict is : fac 
with either a constriction on funds or a much flatter rate of 
growth than they saw in the late 60s arid 7&s» -The, issue is, 
where /do we cut our budget, not, where daft /we spend money on • 
educational programs* So there are some immediate needs as a V' 
school super intendent or as a manager of a school program strives 
to make intelligent decisions in putting a budget together. In 
that case, some of the work that we have done, which attempted to 
respond to those kinds of immediate questions, should be helpful 
initially. " v v 

- There are some- dilemmas that cost analysis can't answer, 
Levin shows us in 1 his book one dilemma associated with cost- 
effectiveness analysis that results from drawing comparisons . , ; 
between two programs. What should we if program A has consid- 
erably more per -pupil achievement than program B, but the cost of 
a- unit of achievement is lower in program B? That is, with a 
fixed level of funding, we can increase total achievement in a 
school district more with program B than with program A. If 
program B is retained and program A /is dropped, then some 
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^children are not given ; the ©pporttmity to a^ievi^ as much as they 
might have : achieved , >bu t nipre^eh itAtmn ; wili shdw^^inodest gain: in 
'achievement* I don't think that this is a problem co&t analysis 
can answer , btft rather one that school Hoards or school admin i-v" 
stca tors -are;/ going to need to look at.v^it's a sod iai question* 
Given the fafet that "resources are limited we vput in the • 
position of having to; make dec is ions * w i thin those contra in ts • 
; ; ; ,Sorae changes could : be, iinpleiden ted without a preliminary 
ana ly s is • : For example , it seems to roe that a^we talk about the- 
'iromediate future and dealing with budget reductiorfcs~the Commis- 
sion on Excellence Report has, been gating a- lot of attention - 
late ly— there are some things that ^an be done that - appear to be 
low-cost items* Michael Kittst' has reported that there are a lot 
of things ! that can be done which don't cost much money. We can 
work with teachers* to improve the amount of time spent on direct 
instruction during the day? wb can ask them i;o assign n*pre 
homework* These are. "things Vthat shouldnVt cost too much money 
and hope fully will lead to increases in achievement and 
performance* ' ; 

There are some other possible changes that ^ are big ^ ticket 
items, such as increasing the length of^the school year ; paying 
teachers fpr 11 or 12 months of servicer increasing the length of 
the school day? paying teachers more money* Those are yery 
expensive, and X don't see any way that those can be done without 
the application of additional resources * to education. 

I have been: talking about two divergent streams. First, as 
we start making decisions, some school districts have tax baaes. 
that are not very secure these days and require short-term J 
budgeting considerations* On the news this morning*, it sounded 
as if most of - the 98; districts in Oregon that had tax levies on * 
the ballot went down to defeat yesterday. So we have a lot of v 
districts looking for money V and more and more of them are coming 
closer arid closer to closure before 1 levies are finally passed, 
That means that districts are constantly ldoking at their . 
budgets, reworking them, making decisions, making reductions. 



right up to fell© last minute , sometline^ after the f iscal -year, hais 
atgtfead* We need some ; ways to ^kW aacisiens % in a short terra- 

• about-^Whera'-'tt " f , ^~ 

Second ; we ; need -to star t look ing at our programs and th inking 
hi/. t^C^I^/biitcdftes ,©r= effects to the \oosts, ;\&o that in the ^ 
future if as ^S^tart doing some long*- range planning V districts fc can 
\ make • g^od ^eoisiphs. that help maximize educational outcomes while 

Fminimizin^ their costs, I think it's very- important that schools 
start toihk in^ ijranedda te ly , whether* it ' s starting a new program 
or \lookirig* : at\i the^ current ones, to plan cost analysis Into their ; 
avaijdatiiQris beginning, if they want to do a good cost 

analySljsV ^ - ^^Z§ ' S::'^- .' J ~: ..: -.. • : "\ ' 

' ! * \ A gQod^cOst analysis may be either cost-effectiveness or 
cost-behe<f It - However, 'both these methods have difficulties 
assoc iated with the ir respective outcome measures. /A- cost- 
benefit analysis, where you attempt to "monetize* everything, is 
attractive because it lets you compare across all ranges of 
programs* Hcw^veFj^lX^s very difficult to do with first- 
.grade reading program. I'm not quite sure how we can . 3e termine 
what the effects on an individual's future earnings are going to 
be whether they were in reading program A or program B in the 7 
first grade. That's why cost-effectiveness analysis seems to be 
favored, \ Cost-benefit analysis may have some uses in vocational 
education programs or college preparation programs where the - 
quality of ; the program may impact the level of schooling/ that the 
person goes on to achieve . I think cost-effectiveness is really 
the one we need to ^Look at . We need to be very clear as we plan 
our evaluation strategies as to what our measures of effective- 
ness are, and whether or not we can keep them consistent from 
year to year , so that we can compare them to costs • • 

We've talked abqu1£ measuring outcomes i the other big issue is 
how we measure costs. One of the tilings, that I* found • in my 
studies is that it 's- very difficult to get good cost data from 
educational agencies, - f I; have a feeling that at least one of our 
panel members will discuss this problem in more detail later, 
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Having served on a citizen budget committee for a eehqoi V.. 
: ^ ;^ dt^telet/; I know, some of the dif f io\iii^^ 

=: \. 1 ve. mlw^W atrug f 1^ wi^ that* Schools traditionally h^ - 
■*V 5 i TOasuirad ^11 of -the true costs of any, g iven program* including"-.. 
; : the program';: ; '",.:-* 

' *'.\y \'y"^«itB..^.\jIt-TOKM'' it very difficult to compare program A^to . 7 i 
% -program B. • "* •*''•""- "V ' > % ' ; * V* ' 1 ' - : : r . <.\ 

I; f o^ind in the cost work I've done, that in many districts, a- 
■ - Xpt o£ people rely .heavily on the •school, administrator to V. 
• determine Aow a' program will be structured- The .» assumptions made 
^boulE^rpgraramatiG structures will influence the outcome of an * * 
- r -Outside- evalua tors - need to *be independent ^ 

'•v.*. • to m^fce some ha£d decisions about what 5 program should look 

like.- The add it ion ^af one teacher, administrator , or: computer a '» 

prog rammer can change the outcome- of a cost analysis "dramati- 
cally , , particular ly over a period* of years. "I've seen in a lot "~ \ . ; 
of our 'work that iff the administrator would like to go a / certain 
way with a program, he .can load* staff here or there to make that 
v happen— to affect the outcome* In addition to knowing what the 
- x - costs are , we .need to be sure that we f re . accurately determining = 
. what the components of the costs are* - \ 

Finally # I f d like to conclude by saying that cost analysis is » 

. '. ■ really only one tool that school administrators are able to use . _. 

in making "decisions. They all deal in a world that looks at many 
factors which influence any' decision being made- Pure cost 
analysis and the making of decisions only on the cost of some- 
thing will never happen in a school district* I understand that 
Beaver ton is considering closing two schools this week. 4 Saving 
money is one important element in that, but as we 1 re' learning in 
Portland, there may be other factors that are more important, as 
we deal with the possible secession of a very wealthy and . 
supportive section of their district from" the district. 
Political and social factors impact any school board's and 
administrator 1 s decisions in the "final analysis, It's very . 
important to always keep in mind that costs are only one factor 

;.,■.• ,.>-• ; ;■ - ■ • V . . 22 - ". ' V - '' 
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to be used in making a dee la ion on what fehe program or structure 
of the school district will look like • Are there any questions? 

Audience Discussion a 

Question t Were you/ suggesting in your ^ribluding 'remarks 
that cost-utility is the most - peeked technique? : 

Answer? No. ' I think |ftat . cbst-e£fectiveness is the most " 
appropriate technique. * However r an accurate cost-effectiveness 
analysis im prob J^Ly not possibly today , because the data haven/ 1 
been collected that allow us to put it together. That's the 
- pcoblenf *we- v ve run into cons is tently V With the results of a good 
cost-effectiveness' analysis, a decisis 

the prog rammat i c Structure should look like. It gets back to 
some of the other social questions: do^we maximize total 
achievement in the district? do we provide the opportunity for \ 
those few children to achieve the most?; I don't think that that 
is autonomous or that I want to make that decision bas^d ~ only on 
cost-effectiveness data. There are other concerns that heed to 
be worked out* The entire school community needs to look *at 
those issues as we 11^ " •. . < . , 

v Question i As I understand it,, cost-utility is the placing of 
value weights on different outcomes. 

Answers kight. Cost^utility places probabilities on the 
likelihood of achievement. For: example, the probability of . v 
program A resulting in a bne^grade level increase in math scores 
is 50 percent, and for program B is 30 percent. These probabil- 
ities ^ are then multiplied by importance or value weights. 
Cost-utility analyses ; can be helpful in making subjective 
decisions, but I think it would still be helpful to really hav^ 
some good outcome data to help make those decisions. 
Jana : We* 11 now move on to the panel. 

Zeno Katterle / 

Larry Pieus "mentioned the success or lack of it concerning 
the passage of special operating levies, I'm still in the state 7 
of euphoria. As one who works as close to a school board as I 



do, "there's one thing that I iaplr ^at more clofely than the : 
results -of a levy or tax base election and -that is the election ; 
o £ board members »^ JftqjstL ^eQRla jrtiss the significance of ;a ;board^ 
election* ^Ordinarily I don' t eoimerit on - iiWividuaL- tK?§f <1 ropjotera 
or candidates* i , ll singly say that the: recent election couldn't 

_have' come but any better from ray point of view* Rational . : ." 
decision making jmy flower, in part as a result of the election* 
It could have been quite different, ; = v ; 

I tried to think through how I would identify some type of 
evidence or examples of cost analysis as practiced by an individ^ 
ual district* The problem I had was trying, to scgpair ate cost 

• analysis~from~ therla^ 

take the perspective of a rational approach to mak ing district 
decisions* There is a vatiety of evidence to» support attempts^ to 
be more rational, whether you put it in terms of program budget- 
ing , being accountable, or just the general concern with 
productivity. l 

' I found it helpful to>look at what the district is attempting, 
to do by using a rational framework, and then f to see how cost- 
analysis/ applies* As a matter of fact, I think that if you asked 
representatives from local school district boards what they think ; 
of cosjt analysis, they*d shuffle their feet and try- to find an 
excuse to talk about something else- They wouldn't .know what - it 
is, to- begin with , and probably wouldn't be all. that interested. 
If you asked the question from a broader perspective, yeu*d find = 
that there's more evidence of attempts to cost programs and make 
comparisons among those programs than we may realize. In terms 
of change , roost 1 of what happens at local district level involves 
three ingredients, One of those, hopefully, is that the change 
is in part baaed on rational decisions. It can be argued that 
the attempts at cost analysis strictly on a rational basis 
contribute information that potentially makes for better 
decisions* It takes a lot more than rationality, and I don't 
think that's aU bad. In addition to a rational base, there 
usually has to be the right, political climate plus economic 
necessity. If you can put those, three things together, the 
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chang # is possible.. I think* in fiaot, that ch^rae fear izes the f 
environment :?Su& ^'V/^V>.:/-1 v v 

: Larry implied : thaty - at - least ;bnv a oompar a t iye bas is , * re * 
dealing with less revenue to do the same - job,, I agree. - The 
expectation is that goveirnment Vill n6t continue to>gr 6w, in terms 
6f vtha lt#l ol expend itures * Unfortunately, y I don f t think thee , ^ 
expTC ^ t ion l asN to j^ia t we produce in the ■ way of service&>is. 
declining. Quite the oppos i te is true . -The- expectation is for 
the level of serv ice . to improve , . Less revenue; and increased Z ; ; > 'J'j- 
expectations have something to do with a heed for greater V / ^ .,U- 
efficiency. So we have the real problem of tyying** to provide an 
improved level of service, with fewer dollars. This situation ry . : '\ 
will continue tor the foreseeable future, toother factor is the A.- 
accountability movement. It is also related to the need for ; ^ 
efficiency* These two factors have done a lot to create a " 
setting .where cost analysis is more feasible, will be' taken more 
seriously, and, frankly, is more necessary for making sound 
program decisions* - V '* ^ : 

With that in mind,: I tried to look for "some' specific evidence 
that perhaps ■ the usetol improved cost information is really 
occurr ing. One th ing - that struck me is that we have an over- 
simplified perspective about school programs,. We talk about, 
programs as though one program is the; same as all other programs , 
with respect to managing and making decisions. — That .is a . * 



mistake, This is apparent when you are talking about cost data,/ 
Support programs such as transportation, food services, and 
maintenance, as opposed to instructional programs such as ^ 
language arts;, and science/ are much more antenable to being ' 
rationally managed in terms of program de fin it ion^what they're 
designed to do and the measurement of the outcomes* " r The use of 
cost analysis is much more applicable when talking about support 
as opposed to instructional programs. Support programs repre- 
sented a substantive aspect of the school operation, and that's 
where, in my experience*. most of the cost analysis work is going 
on . In many cases, it has been pretty good work. . • Jt isn't 



Z5 



ERIC 



coincidental to ^ observations that a number of the ; studies 
cited in JanaT'Sraith v m report %ire support programs r Such as ? * 
\ti»spor feat ion t; .r>; ; ; ■ ; v- .' : " ""ty ■ ^^^^ ;? s .' ••• V 

In niy local districts we've used a process for five* years 
that we call a progran* review, which is .based on technology that~ 

; was rieye loped ^Ith zero-based budgeting , really better described 
as zero-based planning • Our board , by policy, requires -that each 
ye^r^we- look in a very- thorough manner at between 20 to 30 ; : 
percent pi the district * s programs. •/• There are approximately 45 
programs, support and instructional," Part of the rev iew* ii?frolves 
looking at whatekind of a program would be provided at different 

- leve Is- :of ; f und ingv and^ what alternative* str uc t^r es dould be used^ ? 
to continue to apprbx imate or improve on program per formance. V 
Th is ban • t be done very we-11 unless "you clearly def ine and cost \ 
the program and make some assumptions about the possible outcomes. 

It has taken the district about five, years to develop the \ 
attitudes, values, and techniques to mike the review process work 
effectively. In the context of the program review process, there 
are a variety of examples where cost analysis has been required. 
In mos t cases , it 1 s fair ly rud imen tary and doe sn * t focus as much 
as, it ought to on results, but tends , to focus on cost comparison 
or cost-feasibility. In transportation we f ve looked at things 
like comparing the replacement of buses with the remanuf actur Ing 
of buses. We've looked at. d^iesel engines compared to gasoline 
engines. In risk management we f ve looked at self-funding - 
compared^ to premium-based programs ,"; In facilities we ' ve compared 
the closure of -different schools, or combinations of schools. * 
We *ve looked at various energy conservation features in new 
buildings, or remodeling of existing buildings. - We've looked at 
different ways of "organizing and equipping our custodial staff- in 
order to achieve better; results in relationship to cost. 

, In the educational area the d is trictVs attesting to put 
together different scenarios and the associated costs, and 
hopefully /some definition of results with respect to implementing 
a kindergarten program* We will consider a combination of 
contracting Mid operating our owp program at different 4 ser vice 
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levels*,* ife 1 re making various kinds of assumptions \ahout what 
kind of a/ program -will be achievable, given eer ta in limits^ on the 
ability to fund the program* That's proving infinitely ^more 
difficult than it was .with €he support sir vices examples that a! 
prWiou m ly f d i tad* 

Ook: bi the ;firo^iems exper i Weed by the distr ict v.Vis \- that ■ our : ;-y? 
oonstituentavar^rt ^tyail; tKatnwild about our trying out y: Vi ^ ^ - , 

. " ; - ' V-£*AAA :'\ AAA- ~ ; '/VT/\ : V AAA' A-.AA' AAA. AAAA . -AAA- vA '! V A: A -='.. . .AA; '= '■ ■ : ---^'v".t ,A •• A— A-iAA- 

alternatives, tPhe word is exper imeritation , a word you doh 1 1 use • 
whereF I work. Before implementing an alternative, you have to 
albost simulate or. approximate what you think the results are 
going to be and hope you 1 've made the right deals ion ••• By .the time 
a major program change is put into place with all the political 
commitment that's required, you 1 re not about/ to throvj it out two 
or three years later and say %ell , let 1 s go to another plan. " 

.v Whaf are some of the results associated with the activities 
which I 1 ve described? One result is a much better definition of 
program* The cost* analysis aspect forces you right back, to the 
sort of thing-' that Larry Pieus talked about, trying to figure out 
what you ' re doirfg , what adre tjie costs ,.- and what outcomes yOu , 
expect. It f s hot accidental that some . of these x data have been 
hard to come by* * P Educators have purposely made it difficult to 
de fine the programs . and toe costs associated - with them. There 
are some very good reasons for this and in part it has to do with 
survival- Nevertheless,. I think a greater value is being placed % 
on program definition, including the" associated' costs. Oh a 
related basis, there £s a different perspective about; the kind jofH 
cost information that is needed to make choices along with the 
political, social, and* policy issues that are involved* 

Another beneficial outcome has been the willingness to begin 
to look at alternatives. We are faced with a different reality* 
With, reduced funding, it's frequently a question of whether or • 
not we 1 re going to be able to continue to operate a program. The 
only basis on which the program can J .continue is to restructure 
and off er it at a different funding level, as opposed to throwing 
it out altogether . That 1 s a different reality than thsr one we've 
been used to, which was "Hold everything where it is arid we * 11 



just increment increases on top. " The politics have changed 
considerably and r as a consecpiencej people arjin^t willitig to 
accept the incremental approach. As a result of the change y 

•v ',-we -iim : , getting - more * accur a to -ocjs 1 5 alXoeat ion and r ^ese£ ipt ion *; 
^ In add itibnV there is a r ©cognition"; that rather ; than; ^ 

*. incrementing , ^.hivt to lopk at red is tr ibuting fund 3 . This 
^involves taking funds from some programs and adding to others ..-\\ 

, ©lis approach is batter than undergoing a process of gradual 

'•' erosion of everything- That would lead to the mediocrity, or .'• 
death of the organization, I th ink the awareness is there , which 
/means taking dollars away^from; selected programs arid putting them 
elsewhere in order to continue to add and improve programs. 
Though there may be fewer dollars to do the overall job, I know 
of n^^ _ . _ - :; '■■ ; ; . ■ - ' 

V; Another beneficial outcome is a multiple year perspective, 
rather than just "Wha>f are we going to do next year to get. 
through?" That's not unimportant , and certainly a number' of 

r districts are in a survival mode and making it through next year 

1 may be the ii»st .pressing-need to consider , but I think there is a 
move toward looking at programs, their costs "and benefits, over 
more than a single year* : w ; 

My first temptation was to talk about all of the problems 
associated with cost analysis. The first problem I* ve already 
mentioned— the lack Of awareness and appreciation of cost 
analysis. There are a ftumber of others, It's difficult to agree 
on objectives." We don't have very good measurement tools . 
Alternative needs are perceived as experimentation. We haven't 
yet figured out how to clearly differentiate between direct and 
-indirect 'costs. We can't control ms we might like to, for 
outside variables, There is political resistance to 5 the whole 
idea, both internal to the staff and external to the; community. 
There is a considerable lack of technology, skill, and tools, 
You can look at the .list of problems, and say that's a mountain 
we'll never get over. I don't happen to believe it's 
insurmountable • The problems are. some of the realities and we 
can face .teem. v ' 
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What does the future hold for cost analysis? fhii ii always : 
the safest thing to talk about because nobody knows for- sure. - A '.. 
- study, was recently completed by a task force # appointed under the 

" auspices of the Education Commission of the States* One of the .- 
talk £oc€i ^ ciiaonnnendi^ions is i ^"IdUGiloci must ; nfak^bif^t^^^ 
> ;'' -:• of existing resources because no consensus for increased /i]nv©at—:f V; 
T iriertt in education can be built unless citizens are convinced thSt y : 5 - 

schools are sufficiently productive* happen to agree. My 7 

• point is that the need to make better use of existing resources 

isn't going to go away, and shouldn. 1 1 * But there is considerably ==•• : 
;// more work needed to adapt the cost-analysis concept , technology, 
* and tools , to the local school s i tuat ion * % 

... Iii addition, we aire going to see a differentiation in the way 
cost analysis is applied among various programs that a school 
district operates. I also think we'll see variations applied to : 
instructional programs that might be quite different than what we 
would see in support programs. The success of cost analysis is 
in part going to be combined with some other fairly critical 
issues, ; and those have to do with things like how willing we are . .... ., 

to operate with a number of alternative approaches* Selling the 
idea of greater decentralization of authority and decision making 
within ^the school system are also going -to have a lot to do with 
the successful use of cost-analysis^ 

• • / Bob Rath '. v • • •- 

I would like to start by underscoring what others have said, 
and that is that the economic conditions have a great deal of 
\ influence on the way" we look at things. The historical mode is , : 
to start throwing brickbats at the educational institutions. 
That always happens when we have a recession. We get a lot of 
advice; such as; reduce our offerings; don't do as muchi don't 
do so many things for so many kids and people; restrict access; ■ 
raise our standards; reduce opportunity. / Ml of those kinds pf : 
things, so we're in that status at this point. ' 

' " ' '•/ : • ,- ' . . I'"'' _ . O ■ \ : v i-;.. . " O . 
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I Vd like to tell a story about the pig with a wooden * leg. 
This is aboutwthe wa^ education is being treated these ^days. A 
salesman driving along in Eastern Oregon looked out aha saw a pig 
with a wooden leg. He was fascinated because he f d never seen a 
^ig^with^ a^Wobdeh leg , ■ so - he~drove^ into, the farmyard^ arid, said V ; 
; * 1 1 ye; never "seen a pig with a wooden leg - Wou Id yjpu te 11 me .how ) 
that pig got that wooden leg?" The farmer said ''Oh, that Va :k ;;;>. 
wonderful pig— an outstanding pig 7 the ^st pig I ever had. One 
night my wife and I were in bed and the. house caught fire* ^YOu, 
know/ that pig came over and made such a clatter, woke us up, got 
us; put of there and saved our lives. A wonderful pig." The 
salesman said "That still doesn Vt explain how he got a wooden 
leg , f The farmer said , - "Not only that, one time I was down on 
the south 40. The tractor hit a soft spot, went overy and_;I was 
pinned underneath* You know, that pig came clear out there to 
the south 40 and "rooted around and finally got me out. • .saved* my 
life again. -Wonderful pig." The salesman said, "You still 
haven * t explained how the pig got the wooden leg." The farmer 
said fWell> a pig that good you wouldn » t want to eat all at once ! " 

Maybe education is getting treated like that— a pig that good 
we wouldn 1 1 want to eat all at once. Part of the c ir cumstances 
in which we find ourselves is being faced with a whole series of 
recommendations of how to make things better, not fcne least of 
which is to make them better by spending less money. You've seen 
many of the reportg, such as the Commission on Excellence . 
That 1 s a way we get recommendations of how to improve education. 
The data base may be very much askance, and the recommendations 
may be very much politically motiva^ds many of the interpreta- 
tions are. Nevertheless, I think they are based on the 'premise 
that we can improve education. School Improvement Research and 
The Results of School Improvement Efforts are also a set of 
recommendations to us of how to improve schools. Notice, those; 
two do not really talk about costs at all. Some people would , 
infer, as the National Education Association quickly did, that 
the Commission on Excellence would cost multiple billions of ;-■ 
extra dollars from the federal government. Another famous 
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Amer ican ^iiitirpreted it saying fOh , that means we iieed pr ay^y tp^ 
schools , voucher system, and tuition tax^ credits # *^so it = depends' v 
"■V-bii;"'^ ' 
I have not read the report gut out, ^by the Foundation for 5 _ 
Oregon Research and Education, but I was struck by the. news as it , v 
hit last night and this morning • This is almost a quote i v ^ : ;. _ 

/ "•Oregon; education gate a poor gracie. The study says that Oregon ; 5 
schools are getting the - same per f drmarice in . spite of increased ; 
dollar expenditures, * Beautiful statement- If you delved into ' ; • 
the background" of that? from the. little 1 know, I think 1 couldV " - 
/ make* the following statements. First, if you look at how costs 
increased , and they have , the ±eaehers salar ies have probably \. 
increased at a lesser rate than all of the other costs „ Our •-)•'- 
inflation of costs have been in the non-staff cost areas t 
insurance, fuel/ maintenance, those kinds of things. The common / 
assumption out there is that the teachers are the ones that are 
benefiting. Second, we probably have more kids staying. in school 
longer; it's bound to cost more. * ~ v 

^ I have a hunch \that jweJve . installed certain programs where 
the mode of service delivery is mandated and is a v much higher 
cost." I'm talking about special education. I could go to the 
Commission on Excellence and to the Foundation for Oregon 
Research and Education and say "I "have the way to solve the 
problem, If you took off the bottom ten percent of' the student 
population, got. them out of school, then all of these factors on 
wh ich they say we 1 re doing so badly would be changed and we'd be 
better than we were in the last 20 years* " We have to listen to 
these recommendations with a great deal of skepticism. 

Now, more on the notion of cost analysis. I have some feel- 
ings that so much of our cost and effectiveness work has been 
pigeon-holed into what I would call organizational dichotomies, 
and perhaps tills is the *first place we nave to look. Some of the 
places where I go, the financial people and the economists are 
called bean counters, We also find evaluators set off by them- 
selves, outside of the main stream of the organization. It seems 
to me, that one of the things that we have to deal with is how to . \ '■ 



get "those two very needed functions , that is, cost-analysis and 
, measurement mnd ^reporting f into tee organisational mainstream Bf 

the iahosls, , I ein't speak about hospitals' or other places , hut 

think it's tf ue about schools. To do" that, we! re goii^j to have 

to -examine' the f uSit of /the " tfnaiysis and::- the" unit of s ^ s V ; ; . 

accumulation, Host o# r fel, effectiveness work > ; that is # • ; 

. evaluation , has begun from ■ a .... top _ dawn -demand , from the federal .- 

government , and we've all been very carefully responding as much; 

as we heed to to satisfy those federal responsibilities,: ; ' 
Second, it seems to me that the unit of analysis has pr imar- 

ily been in a schcol dist^rict- _ If you ask what the budget or .] \~. . 



expenditure is, you can always find put what they are for a 
school district- You can rarely find it at a building level/. It 
is ^pos s ible > ttiough ^ You can, at times , f ind_ program - de fin itiohs 
and programs can be utilized as units" of analysis • and 
accumulation - It seems to me that with our current technology 
and thinking, the most logical and promising unit of accumulation 
is the building level, and I would suggest that this would have . 
the greatest- promise foic 1 us to look at cost and effect—at the - • • - 

building level* There are going to be a whole series of things 

- ' .. - ' • • : / • • , : " " . - " .-. .>-- • .. 

that muddy the water because, as we all know, the variability in 

costs tends t6 be* almost 100 percent variability in just the 

teacher's salary alone, so you have to look at it" carefully. 

Nevertheless, I believe that school-effectiveness research is 



best implemented at the building level, I also think we can " • % . 
adapt" cost— analysis to a building level and come out ahead. 

What this probably means is pretty careful negotiation of a 
set .of definitions, tad I'm not sure that cost-effectiveness and 
cost-utility are the best set- Maybe we have to do some simpli- ^ 
fication. Maybe we have to look at what is the plan at the 
building level instead; what - are the budgets for that plan at the 
bu ildi ^ , leve 1? Then document what happened as that plan wa s 
implemented, and what it cost at the building level, Then look ^ 
at the outcomes at toe building level, , All of this is completely 
feasible at this time, and we can begin to look at the building 
level as the unit of cost and effect. .1 think we all know that 

.' .V ■ ■ % V ■• '. - '•• 




_ it's the building* level that is" tha^ fauiiding >ioak ft>r buflfit^ \. : \ 
. elections, jwhether it be- school board member or budget elections • v - v 
But tti is negotiation -:is of a set of coraaon terms that can be used 
year after year as a data base, not a study base, and not as just 
an infoirmmtion base/ but rather as a da ta' base at a ^ bu ild ing \ ; 

| ' leve 1- tost analysis is feasible and can be done, but hash 1 1 * - ■ r 

/ - been done very we 11 yet . Effectiveness analy s is can also be /. \ ; . v - . 
doh#> and' tyick. Smith', in his program, has led us to new ways of 

:[ : \, doing effectiveness analysis- If you can do those separately, at 

the build irig level, we can make sense out of It - ' : 



: : Don Egge '-:}' ' • - .- • .. . -.- '-,;■> v".-. 

I thought a little about a title for this session- One was 

- wp 

An alternative I had was "Creative Cast Analysis, or How to Kill 
a Bill by Referring it to Ways and Means." I could go on,.. 
Jana Kay suggested that we give some of our own experiences', and 
then talk about the future. As I thought about that, I remem- 
bered # as a high school principal years ago, my inability to make 
a case for the needs of our school, I went on to graduate 
school, taking the usual courses in economies, pbblic finance, 
school finance, etc., as part of the administration degree, and 
still I was incapable of under stand ing and expressing fiscal 
; needs and impact* , 

My first year in a central off ice was as an assistant 
super internment in the Lincoln County School District, a district 
.» consisting of ten small communities. The district was struggling 
financially when the hew team of adminfistrators arrived. Within 
a few months the district faced the difficult problem of what to 
do with ah aging bus fleet that was totally inadequate to handle ,. 
* the half-a-million miles to be covered each year. Our business 
manager produced a single document, fairly simple, straight^ 
forward, but superb, which was presented to that board. On the 
basis of seeing the history of the problem 0 we were facing and the 
projection of the future, the board made a 'timely decision to ' r 
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S^eJi^he bu^es ; and get into a leasing * pr^ram^ was the 
correct decision for that time* ' • : 
' . As^a superintendent a few years later , I remember two = 
incidents that really came tyme to to. We had two situations t 
(1) we- had a grandstand burn down at Newport, and (2) we had a 

1 locker room in S lie tz which seemed to constantly have iSl, inches * 
watif in*, rit/r u , Justly on the girls* side. As the school hoard 
started t©: deal with the two issues (remember , each of the X 
principals in those ten small communities worked with the local 
school committee , hu ilding up his own constituency) , each problem k 
was joined by v every other problem those principals wanted to 
solve * The local school committees- got behind--them r -and we had ^ 
what could have been a very difficult confrontation between the 
board and each local Because we we re able to ^ pu 11 - the _ _ 

problems apart and provide an analysis of the individual 
sub-problems with in each issue , we were able to avoid a very 
difficult time, with a fairly high degree of satisfaction of 
those involved* 

X work at the state level and \we have a different set of 
problems from the kinds Zeno was sharing with us. We face both 
macro and micro issues. We have a budget of $1,245,000,000, We 
must come to the legislature every two^years and try to provide 
options for : that body, which will result in good decisions. The 
process is not always rational* It is often said that when the 
legislature is in town no one is safe- If you have been watching 
what has been happening in this session, you know that it is 
indeed a very irrational process, Nevertheless, we do go through 
a series of steps that start with very small pieces and which 
grow and accumulate to very large kinds of decisions. * 

When we speak of grant in aid, i,e, , distribution of payments * 
to districts, we mean things such as a special- education formula 
for distributing money to regions, to districts, to serve high 
cost, low incident students. Or, when we were assigned to 
administer the new block grant, we defined a formula for distri- 
bution to local districts recognizing both student enrollment as 
well as high cost studies. * 
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basic -school support, community college operations or any of the 
other categorical programs, we provide analyses thit' hopefully 
help the eotoittees cope with those/ tough V dec is ions. .. " 

At -the state levels aliBQSt * every issue has a ; fiscal dimens iori. 
Our bierirt iai; budge t a f f eo ts a 11 segments of eduaation, To obtain ; 
apprwal, we go through a lengthy process s t (1) reviewed with the 
state board, , (2) reviewed with the governor f s of fide, r^jpwed 
' by/ the Oregon Education and Coord ina t ing Coiranission, (4) reviewed 
by the education, subcommittee of ; ways/ and- means, (5) /reviewed by 
the full Ways and Means -Commit tee , and (6) : finally, - by the ,/-/ ■\ J? .'. ; ; 
legislative body itself. If anything can survive that, in any 
form of rationality, I'd be surprised. Every budget we submit, is 
based on a six-year "plan : the last full two years j the estimated 
t^o years- of the -current biennium; and t^ 

We're trying- to make cost projections that effect local school -, 
systems based upon - where things will be in 36 months, or even 
longer- - • 

. As priorities are identified' and supporting analyses are 
prepared , the state ; board is pulled between and among constitu- 
ency groups that want more money for talented and gifted, the 
handicapped, child development specialists, vocational education, 
and the disadvantaged. At the same time , the board i£ aware off 
the views of the governor and the legislature, The board has 
come to the conclusion that a budget plan must be presented which 
meets the needs of the education system of Oregon, even though it 
may not be realistic in terms of e the funds available, " . 

Within the agency, we develop a variety of cost analyses when 
we have special problems, Some off those speeial problems affect 
districts, for example, we might have the Secretary of State 1 s 
office telling us we % must change the indirect cost rater we must 
pull federal money from programs and use it to fund department 
operations. To accomplish th is assignment, we must understand 
the balance of impact on the federal program on our agency 'and on 
pur distribution to local systems. *< - 



r ; I^have tried to look at olir topic at the polio?, management 
and service 'levels.' At the service level r one of our responsi-. 
bilities istt© help> local schboi d istr ic ts oope with; the cdst 
analysis or management information problems that 'they have, Yotr 
may be ; aware of the managing-cost-creatively pro j ect we ; conduc ted 
a year ago f in which we tried to collect and share ways to T 
conserv^ re sources. ■ For example * we referred districts to other 
districts which had compiled cost-analysis* We shared the 
Beaverton study on die sel versus gas buses , . the Ashland study on 
new approaches to roofing r and the process that: the Portland ^ 
district used in managing cost reductions. One of the roles that 
we have is to share that kind of information, := 



A second responsibility is to- collect information from 
Jflistric^ federal jpur^^ 

trying to learn is to develop our reporting forms in such a way 
that the information given to the state by local districts can 
also be of use to the local districts, : 

At least one program in our department has taken the informa- 
tion given to us by local districts, transformed it into ; " 
management information r wh ich has been fed back to the distr ict • , 
You may be familiar with the work in the food services computer 
program that translates raw data into a printout for the local 
; food services or business manager, The report describes -some of 
the relationships between costs and outcomes* such as meals per 
labor hour* Our food service' people are able to circle factors 
they want to call to the attention of the local food services 
manager , following up to provide assistance and training for 
making better decisions. 

We were asked to comment about future, possibilities. My 
comments are more reactions than predictions, ; We must remember 
the difference between research and evaluation, I'm not a 
# scholar, so I can't deal with some of the refinements; but rather 
, than looking for truth, what we are looking for is information to 
help us decide and act, As a manager , I am willing to use my 
judgment, my experience, and others to help me examine the, 
information we have, and to translate it into good decisions, 
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The issue before us is that we new have very little management ; 
information available to us* We have extensive data and reports* 
bu£ not, very mudi of : it ; has been translated into management 
information . However 9 I believe we've seen over the last decade 
an inorease4 willingness and ability to begin to create and 
report management information which is simple , direct and timely, 
■ ^ Data are not Always inforination*, i : A little information -ma be 
better than none, at alL. We. may not need precision, but what we 
may need are trends, in terms of effectiveness, productivity * 
impact , etc - A rule-of-thumb might be to obtain ^the absolute : 
least amount of information needed in order to make a decision. 
In order to do that , the manager must be clear in expressing to 
the analyst what is needed. Jana's premise is correct that most 
folks are not prepare 

We * ve heard some comments about the heed for cost-effective- 
ness . I "believe * we ' re going to learn a lot about the subject 
over the next 15 to 20 years. We may waver between two extremes* 
(1) therte are complex, highly sophisticated reports that may not 
be of much use, to the manager or policy maker j or (2) there may 
be simple displays which allow the manage? or policy" maker to 
review information sets when considering options, The second may 
not meet the usual tests for validity and reliability, but the 
manager is often willing to accept less than what we often think 
is desired and needed. 

One of the things I've learned is that there are some things 
I can control and some I can* t. When you talk about time* you 
have *a state board that says you must have ITS days of schooling. . 
b You have a local board that says how those days are going to be 
organised, You have a school, that says when toe period will * 
start and end, and when the lunch hour will be scheduled. T?nen 
you have a teacher that 1 s really responsible for most of the .-. 
academic learning time decisions when the door is closed. We 
ought to learn how tq deal with and recognize^ those things that 
are controllable, and those that are not controllable. We have 
to respect the role that each is to play and how we can use, that 
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to help make decisions at the various levels. We need to g iye 
each- the . proper information . • % % *• 
- > Secondly, I believe there is a lot of readiness far cost 

analysis* During a period of fiscal constraint and 'return to* " 
conservatism ^; people willing : to look at th ings ift jr differ #nt; • "• 
■ - way . : itou don 1 1 r ea lly know whether d t 1 s because they 1 re 'com- , ; v 
. . • mitted to good stewardship, .or whether they just want to survive, v - 
; We know,* for example/ that of all of the superintendent fir incfs', .<■'.'. 
\; in the last couple of years, the root of each dismissal was j the \ v 
inability to manage costs . I believe the general readiness is ; 
.' = , real. * *• t ^ v \v- • ' - " : " ; -.\/ : "7- 'V. " - : 

; Cost analysis can help us to make; better decisions, I would r_ 
challenge one of the statements made earlier in th is se s s ion > T 

I believe, we must. We may even want to return some of those 

, . - . . * - ; .j .. ...... ... ...... - . : . . . .... \ . t . . . •, • 

resources to the taxpayer , or at least broaden educational 
opportunity when we find new efficiencies* 

Finally, educational systems are complex social 
organizations. They're not always rational; goals are diffuse, 
These tools ought to help us identify and describe educational 
problems while understanding the impact of potential solutions . 
Let us continue our search for improvements in our capacity to ; 
- i : * gain -useful information. " ■ " 

Audience Discuss ion 

Question i I'd like to make an observation and perhaps get 
Don's readtibn to that observation* I really appreciated the 
insight you imparted with respect to the decision making process 
and how different kinds of information are used in that process, 
-.- I think that one of the deficiencies in evaluation is that it is 
a professional, field, You distinguish between evaluation, and \ 
research along these lines, and most ©valuators, especially the 
€ young ones, have this desire to be elegant and sophisticated ^and 
reflect their research background in the approach that they take 
in conducting evaluation* studies, They. pay lip service to the 
| idea that the purpose of evaluation is to provide data for .. 

■ - ' .-. .38 . ' ; - ..... ; . :. 



decision making, but' Very few of them have;; a vary good grasp of 
the nature of the decision - making process ^tha t = the^f r e | ^dlh§ • . : - 
data -lntoV;''^iokl7f/'TOme : "6£ the charaoteff isfeios of that - f as I > 
ifear d; yoa int ! thiia> ara iaioipeEattay cQntixtJspinaant, --^ 
complex in nature/ and very d iffioult t©^ model, Xou~ean • t reach ^ ; ^ -V 
out- and grasp a particular dec is ion -making model and assume that, 
r it" is in use. Maybe we need some mechan ism that ^would bapt i2# • ^ ; 
^valuators into the ho ly water of decision making and drown a few. 
Answer £ You really captured 2 a concern I have* Most research 
can probably help us in evaluation; it can.be .useful for evalua- V 
tion and ^decision making. Probably some evaluation , could be / 
useful to research. r We do need to be clear about the difference. 
I - like your summary of the complexity oA^decisipn making. * I made 
the-TOimneri^ 

would like to say Jthat X may not always know ,what I need, so ^ 
there must be a communication and negotiation between the 
decision maker and the evaluator.y Xt is important -^that they 
continually touch bases to make sure they are together on the 
issue. I believe one danger is that you could put on." such a 
beautiful display teat everyone would assume all variables were 
included and not give further thought beyond the report* I don 1 t 
think decision making and policy mak ing ' should ever be void of 
experience/" judgment, timing, and an understanding of the- - 
variables. However ,'. it isn f t possible to include -all these V 
factors in a cost analysis* But. hopefully, the decision is , 
better when they are coupled with good information. 

Walt Hathaway . 

How toat we* ve listened to ^eno^ and Bob's and Don* s 
speculations and observations on cost analysis and education in 
general* I'd ii*ke to make a hard-hitting, data^based, scientific 
presentation on the specifics of cdst Analysis in educational 
evaluation v Unfortunately, I'm not prepared to do that. 1 
therefore will echo ai£l perhaps argument sentiments of the need 
for a common sense approach to approximating attainment of the v 
vision that many of us had through the mid-60s and early 70s 



about supplying econometric, models and processes ! to education and 
i coming tup .vritft true e^st^f festive modeling for inforniedv 
dieisiosni in education. And^ I'll try to do that from a local 

;,pftrspieartlve',;; \ : f ' r • ^ v , : .v. '3' ■• ••' \-.\~: 

- ^ the sad state =e£ cost^ - 

"Mdlyaa^S/J^ give some evidence for it # 

deacr ib^es is the case* I will then talk about the concept of 
^St^^^feetiveness and the historical and theoretical problems 
that I. believe have > impeded our progress ,in making cos t^ • ? i > : ^ 
eff ectiL_ veness a central and : effective aspect of educational 
ivaluat=ion. Finally, I will describe some of the things that we * 
ire doL_ng in the* area of cost-effectiveness and evaluation in • 
Por t larm<a "as a ^system ind'~ as a depar tment * In clos ing , rl w i ix- - 
make a oouple of very non-revolutionary suggestions . 

Thfe= hypothesis is that in spite of the lip service paid to 
cost program evaluation - f these studies have not been # And are 

not tyj^- ically now, -a,; pr imary concern of the school 'district 
evaluate n on un it s , When /costs have been a concern , they have 
usually - not been competently or comprehensively de£lt with, 
f fivjL«*aence for the above hypothesis is to be found in the v 

, literal- %ire that you all monitor and are aware of • When was the 
last tL^me you read an evaluation that dealt with the cost and 
evaluate ion and linking of the two in *soroe incisive way? As I 
reflect back on the last ten years or more of meetings oi; 'a.- - . - • .• 
national ZL group of directors of research and evaluation that I 
attend ^each year toe day before the American Educational Research 
Assoeia.^fcion annual meetings f I find that rarely has the issue of 

, eost^arL^a. lysis and evaluation come up at these meetings designed - 
to shar^e concerns- Similarly f Division H> 'which is the "program 
avaluat^ion and development division o£ AERA. has for some years 
now giving out awards v for the* best work in evaluation around 
the .couintry. Hundreds of people doing evaluation send in 'their 
^ist'^p^ck ... for. review,- I chaired this process a couple- of years 
igo .'and, have monitored it since* I looked over: / as part of our 
pfi^ara ion for coming over here, . the list of th is year f m awards; 



Eyen in the category where you would expect to find something, 
"Findings from a Research \^Study> ofi^a^Meuiagement. Related itemV" v 
the best reports did not indeed deal with cost in a f or thr ight or 
par t iertcnr ly accur ate way . , There havti , howeve r , been exceptions 
: ta that in the past and I will aite some of those. ' 

- r : Thmrm. atfe two school systems that I knbw of , and i ■m sure ^ > 

[. ■;' : . : the tm ■ -■'«(* *'•■'.* others, who have been "trying to deal creatively andl ; r 
^: : ^:-yy;rX -XntppMi^iY^^^'-'^^ issue of costs system-wide* and relating 
^ \~\ -leasts to evaluation* , In the Montgomery 'County School pistr let f V ^: 
, where Steve vFrankel has the accounting function under him -as Well . 

- as the functions that are more traditionally, in R&D unitsy$ there 
; : . has been careful arid effective attention to cost analysis in 
" evaluation f Wh ile Larry Barber was there , the Eugene School 

^ vf District also worked in this area effectively *; They* did some: . ; 

to the system. These' studies had to do with such things as 
■'. different k inds o f » fuels arid choices be tween employment of aides 
tod other iiistr actional support personnel. So it's, not impos-* , % 
sible to dp^seful work that deals wito tHe relationship of costs, 
to outcomes^ and informs decision makers in a timely way. Such 
work affects decisions that trim support budgets and thus f ? ee _"P 
nore resources to affect kids directly.- : 

The last thing I did to check out my impression of the 

impover ished state of cost analyses and evaluation was to look at 

. ,\ • - * • . ■ . . • **: . ■ ' "... ' - ' _ ; ■ . ' .■ ■ ■ ■ 

a publication which some of you are familiar with, which is % 

: Research and ISvaluaton Studies from Large School Systems, an 

annual ERIC publication that funs about two years, behind current 

^practice. The only entry here (put of thousands) that comes 

under the heading of £ost-eff ectiveness is a cost study of year-* 

; • 0 round schools that wa% primarily in accounting, Tfcqs, we . / 

v basically find one report out of thousands of research arid 

evaluation studies ^>ich were submitted by large school systems 

that explicitly deali with *cost. I think we can take it as the 

fact that riot a lot of wbr k , or at least not a lot of good work, 

is being done in thVs" area, but that there are examples that show 

that it is possible. \ \ * 
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h One 'txplfhafcion' for why we in evaluation units have not been 
taking the lead in getting such work done includes the- fact that 
mast evaluators and ^directors jot evaluation come , out of art 
'educational gsyoholbgy^ot^ an eduoat ional administration baok^ : 
ground - 9 where there has been little concern for cost.. They 
didn't teach jne a \Aole lot about cost analysis when I was study- 
ing educational pschology measurement and evaluation at the = : * 
University of ♦Pennsylvania - But earlier when I was at \ Penn * s 
Wharton School of Business and Finance v doing a ; dual major in,.;.,:' ; -.,= ,•'=■' 
systems eng ineering and operations research with the Mpdre Sphool 
of Engineering, that's about all they ever talked about besides 
circuits*- ^ ; ' , ";„;...; * . ... s • '■ \ ../. : ;.J SJ ]\;J; 

Another explanation is that educational research and / 
evaluation units are usually aligned with the instructional and 



direct instructional support units, both philosophically arid 
orgriizationally f in terms of mission and goals , rather than with 
the business side of the organization* That's pretty much an 
unknown territory- Organizationally, we hardly ever sit down and 
talk to- the business people* • We don't even know what' they do, __ „,L 
and thatfs where cost can most readily be dealt with, as Zeno was 
pointing out. This doesn't mean that instruction <3pesn',t cost 
money- It just means that we don't have as many people well- 
drained and, conscious of cost on our side of the street. All you 
have to do is look at typical school district organization 
charts, and you will see that this is the case. i 
• Finally, the concern for costs in education has been linked 
with econometric modeling and with the development and applica- 
tion of the Program Planning and Budgeting (PPB) method that had 
its heydey in the raid-60s and early 70s- This was sparked^ , 
partially by some international concern by UNESCO, and the " 
Organization for Cooperative Educational Development that did 
have some success in using cost at a national and international * 
level > and doing some policy planning. However, there wasn't a 
let of follow- through in the attempts to apply econometrics, and 
not a lot of success and rewarding feedback from those attempts,. 
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so as we rooved into the mid-*70s , a, lot of us who had been 
involved In that-got very disenchanted ^ith it* ^ 

At this point I want to quote what I think is a very simple * 
and yet useful, definition of cost-effectiveness dialysis to ._. '■. 
illustrate why I : think , S^e * re , struggling getting tp this idem of 
cost-effectiveness, and where we have to be instead.* This is , 
f rom Mark Thompson 1 s; book on benefit cost analysis for program 
evaluation V which doesn't apply solely to education, but is a 
good survey of the applicaton to social science in general • 
Mark's definition- of cost-effectiveness analysis is "evaluating . i 
decision alternative, first of all by making all aspects 
co-measurable in .money terms, or in terms of some other single 
unit -\ w ; The keynote there is co-raeasurabilty of outcomes. ' - 
Really , > really tough . Measur ability, maybe . Co-measurabil itv , , 
probably not. And then comparing the two dimensions of impact? 
That's exactly where the dilexnna is. Can we measure either cost 
or benefit? How do we de ve lop useful co— measures? That is , how 
can we reduce educational outcomes to some kind of coimnon metric 
qf money - or .another underlying metric? 7 

I want to illustrate how little I think the. field has moved' 
in this regard by quoting something that I wrote back in about 
1969, which" was entitled "The Use of Operations Research and 
Systems Engineering for Rationalizing the Decision Process in 
Educational Systems /" This was going to be the grand synthesis 
on that subject. I stopped when it was 80 percent done because I 
was sick of it, and I realized I didn't really need to do it for 
the degree related reasons I was v dping it. On page 74, one of 
the points that I made then was that one reason why true explana- 
tory or econometr ic models do not yet exist in education falls 
back on the , by now, familiar problem ^of the ^tack of an explana- 
tory theory of learning * This gets to the benefit side of the 
dilemma. If bur dependent variable is economic well-being, and 
our independent variable is dollars entering into a school 
system, nd true explanatory cost-effectiveness, model can be built 
without two submodels i one that yields the effect of funds 
transformed into teachers, materials, and space in the learning 



process, and another that yields the effect of that increased 
learning on individual and societal economic utility. To put it 
succintly, we don't even know how to measure learning, and there- 
fore , it 1 b very difficult to pinpoint comprehensive relationships 
of benefits (learning) to costs* • „ • 
■ • ; f^inally^ let roe g ive a br ief descr iption of some of the 
^things that we 1 re doing in Portland in the area 'oftoost*- 
effectiveness arid evaluation * The f irs t th ing XVd^ say at th is - T 
point is that cost remains frustratingly difficult to measure 
accurately , comprehensively and co-raeasureablyv in educational 
Organizations* To give you an example § we ' ve been doing an 
evaluation of computer use £n the schools in Portland for two 
years* for which there is ah apparent budget allocation this year 
of almost three-quarters of a^roilliqri dollars. The "real" budget 
a lloclit ion , however , is that amount* plus whatever building 
principals and- program leaders are allocating from their Chapter 
1 and other discretionary sources of funding* in order to deal 
with the pressure that they're getting from the community and the 
sta f f to have computers . The agenda is : equ i table; hardware r ; 
-dis^rtbatiohT— ^athee^than effective learning at this point and 
even at that the budget is difficult to ferret out* much less 
real costs and the benefits* 

Still on the cost side* but getting away from budgets* we 
have costs that we could objectify* like the cost of hardware, 
the software i| and the staff development , but then there are 
hidden costs and opportunity costs. The opportunity cost is an 
economic notion that we have to deal with when we deal witii costs 
comprehensively. The opportunity cost represents other things 
that those dollars are not being spent tor, like additional 
teachers* library aides, and tilings like that. The budget is not 
an accurate mirror of the reality of what is being spent, 
although we a lways war k . to try to make it so . . So don 1 1 ever be 
deluded when you 're trying to do an analysis of costs that you've 
got an adequate handle on it when you have merely the budget 
figures. . v v . - f 4 : \ "7r: . :.: • 5., 

■ ~ • 6 ■ ■ ■ ' ' ■ ■■ ■■ - ■. ■' ' > ■ 



Benefit, or effectiveness, as^XJve already said f is even 
tougher to measure* ^r instance in an ace© like Chapter 1 we 
have short-term and long- term pi r sonal, system and societal . * 
benefits, We typically f ecus on a subset of the f^rsonal, short* 
term benefits, which is achievement gains. But what about the 
changes and gains for: the system /and then the changes and gains. 
,for Society #1 both ; short-term and long-term? ? I*m thinking of 
things like long-term health consequences , impact on or ime rates 
effects on otherwise* disadvantaged children, and effects, on ii:^ # 
numbers of people on welfare* ;;v^^v =;.v 

My - conclusion is that if costs are difficult to measure in- 
accurately ^ comprehensibly, and co-measur ably, benefits are 
almost impossible to, measure at this point in education . Some -of 
them are even difficult to define j that 1 s, why we find cost- 



benefit or cost-effectiveness analysis difficult to do. 
: y In spite of the gloomy picture painted above as an evaluation 
department in Portland, and as a district, there v are three flings 
that we have been doing that show concern for and some progress; 
in the. 7 area of cost-effectiveness in education. First of all* we 
bring our skills as e valuators to the district budget making and 
communicating process, and also to other cost related policy 
arenas such as school closures and consolidations. Reports of 
these activities are to be found in documents such as my article 
on Managing School Districts in a Decade -.of Decline . This report 
documents how we went about using objective information related 
to programs during a two-year budget reduction process so that we 
could use cost-effectiveness information to help decide what 
programs to eliminate or reduce when we were in our last major 
period of contraction - 

Secondly, as a district we also occasionally do evaluations 
primarily f ocusefi on cost comparisons. I'm thinking f0r instance 
of a study that the transportation people did on the relative 
utility of propane versus regular gas. The study resulted in a 
conversion of some 50 vehicles to' propane, with a savings to the 
system of $100,000 a year. Another study led to the use of 
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computer i?ed transpqr ation; assignment Bys terns > , J using a^prithras; 
'^pmijfig out of the operations research simpler method. This 
effort is resulting in a savings of over SI ,000, 000 a year 

r -jjn reebghition -6 And; co§t studies^ 

are the direction evaluation departments must take in the future 
in order to continue to justify our existence, I recently hired 
somebody from the,/ risk inanagement side of our organization , which 
happens to be the business management side/ as an evaluation ' : * 
specialist - Right now he has another assignment, but one of his 
long-term goals is to ; build better bridges between his former 
colleagues and 1 us. That's the way to do cooperative anything— to; 
br ing in someone from 1 the other, side , or else go over ^ to the 
other side yourself, v - 
V • . Finally, as a department we do program evaluations which : 
treat the -issue of relative costs descriptively. For instance, . 
in our evaluations 6f alternative schools, at least we have tried 
to get at the per-studeht cost and the per-unit of instruction 
cost of each of these" various education programs. We 1 re also; 
trying look at the cost-benefit side of things in our third year 
of evaluating computer use in the schools. 

Those are examples of at least the kinds of cost related 
things that we get into as a school system and its evaluation 
unit, and our struggles to, deal effectively with costs and 
benefits, The last thing I want to say is abbut Jana's report. 
1 thought it dealt well with cases and reports where costs were 
the primary focus, and I especially appreciated the fact that the 
cases did deal well with the merely descriptive stuff. Before we 
neve* up that hierarchy, our goal roust be to again describe the 
costs accurately; comprehensibly, and co-measurably, so that the 
cost of this alternative program is in the same metric as the 
cost of that program y not what their budget says, for enample. 
And thenv having done that, we can get into feasibility, compar- 
ing those co-measurable costs with a co-measurable limit, and we V 
can get into utility, effectiveness, and benefit. The first job 
is just to work on good descriptive techniques. The development 



d£ «r wdi kbook ^ a: woe kshop j or '# = helps X to - that end would be ^ 
appreciated by local, and/ I fehinH f state systems. ^ 

Of greater interest * perhaps* would be a study of the use of 
eost-ef f ec t iven in program evaluations and policies analyse s*- 
to loof^ at th0»*^ where obst : was.not the pr 

where it dealt - # with helpfully evaluation we don 1 1 : 

liitfemally priraar ily set out to evaluate costs* What we need to ; 
learn to do is just that; we need to learn how to deal with co^ts 
when our main job Isn't to dial with easts, Again, I think a 
good source of that would be the general literature* the American 
Educational Research Association D i v is ion H awards documents * the 
R&E studies of the large school systems (in ERIC) and other V 
sources » The study could provide examples of A what 1 s been done , 
and been done well 9 and then extrapolate some models and proce- 



dures for doing more of what has been successful* * . 

Some of my conclusions are that as eviiua tors *' we have a - - ^ 
moral and educational and probably survival obligation to at 
least describe costs accurately * comprehensibly # and in compara- 
ble metrics* We. have-to realize that that is not easy_ and ^we, \ .. _ . 
need to work, at the same time, to extend the range of projected 
measures of educational outcomes and the range of less objective 
measures* It is pathetic to me that most of our decision making 
about effectiveness *-- when we do it in my school district, and 
probably most other places* is based on measures of basic skiljLs 
achievement* and that aflone. This seems to be true of the whole 
body of research on educational effectiveness, particularly the 
effective schools research* We as evalua tors simply must begin 
to move more aggressively and assertively into creation of better 
and more standard measures of effectiveness outside basic skills* 

There is a regional cooperative being mounted in Oregon now* 
Kith leadership in the state department in the area of science 
measurement* that I think will be a good model of how we can move 
into other subject areas* There has also been some work in 
higher-border thinking and problem solving done by our district 1 s 
gifted -and talented people. By cooperating with the state 
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department In soma other districts, we went outs'ide traditional 
subject measurement end into -other, k inds* of , measurement* 

{ Only after we've figured out how to measure easts aeiurately, 
and after we have .better and more Comprehensive measures of 
effects, can we , worry about interrelating them into ooet- 
effcMfcivenesS;^^ ; ' /.;->(•'.- y. ; J ;y ^ .>^';'-=^ :=.: : -, f ... ~. 

General Discussion - * - * 

, Question ; First of. ail, I'd like to thank all of you for : " 
participating on this panel, A comment that I'd like to make: 
Bob mentioned that there are two groups of people . I think I'd 
separate the economists as another groups as they seem to be a 
separate group, besides- the accountants and ©valuators, . The 
economists are the ones that work on methodology- As one looks 
back over several years, it seems to me there has been a big 
evolution from the evaluation side, from the educational 
administration side, and from the accounting side, in terms of 
looking more at the costs and at the effects- . .The things that 
^ hay^Vt change 
methodologies of cost-benefit analysis or of cost-utility 
analysis are the same ones that were written about 25 years ago. 
It seems to me we need help from economists who will be a little 
more sensitive to the fact that their methodologies cannot 
necessarily be held up as the ideal if they don't fit, > We need 
some help from them ^about either how their .methodologies can get 
better, how they might change, or something. It would seem to rae 
that some evolution is needed there, as well- - * 

Answer s I don't personally see anything either in economic 
theory or raetnodo logy that cou Id not be, applied appropr ia te ly to 
decision making in education at various levels, What I see is 
slof^dhess of definition, sloppiness of description, and poor 
measuring that makes it impossible for us to use those theories 
or measures- That 1 s* where I think, the real task is, Dollars are 
easy to county and learning is hard to count. 



V Question s I hear the review af the work that , has gone on? I 
see where the existing methodologies apply in service areas where 
" ,*>■ -i yau can count bodies trtopj^^edf? But j you don ? t 5 * have the ; 

- .industrial contacts in education, so there are just a narrow band 
of places where the apst^Malysis methodologies a^ly; J i think ' 
it's me thadalag lea 1 weaknesses as much as our failing: to live # up 
to the standards of assumptions beh ind; tlie methodology . . - r 

Answer i As you were. talking about methodologies, . the thing 
v~ ; that .came to me , f rdm the po int-*of -v iew of the manager , is that X 
would really like to know about product options" of cost-analysis 
'7 7 efforts- . If there is something that needs some attention, it is 
perhaps a handy-dandy notebook of information on products. An 
analyst brings~"some thing and you say "No, that's riot quite what I 

it's. a slow process and as a result, the methodologies have to \ 
get changed, It seems ;to me that you should be able to -describe-. 
th£ form and way it would be delivered. Now, given that final 
product, the evaluator would know the alternative methodologies 
that to ahead. My plea is that 

you shouldn't just 'think about the methods, Pity thte poor user. 

Answer : I'd like to take a stab at the question, X „ wonder 
if one of the problems wi.th what's happened is that the bulk of 
economists' attention in the .area of education- has been focused 
*V on the school finance movements how we distribute funds to 
schools, rather than how schools spend those funds. A large 
v piece of that wor k has attempted .to prove that if you spend more' 
money, you get better results, largely through obnoxiously large 
and incoherent productivity fraction equations which try to show 
that one of the important factors in student outcomes is addi- 
tional expenditures, What we largely try to do^is take a model 
that works when the outcome is dollars and apply* it to something 
where the outcome is people and abilities, or learning. To date 

there has not been a lot of success in attempting to .find some 

;V V< ' v ±% - ■ " : . ' ' . . . 

9^her measure that works well, One of the problems is that a 

; - 7 dsSllar is a dollar *minus inflation. People start with differ- 

" ij^ent levels of ability and are impacted by so many things along 
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the way that it is vwy difficult to -compare wha€ I got out of 12 
years Of school with what my next-door neighbor did^ because of 
v so many impacts that may \ have affected us. Methodologies haven't 

found ways to measure and control for tha^, . \ - 

> ^ : Answers Wf are, not a production industry, y Wte ire a: service \ 
;•• V' ihdustry . A concept of productivity has merit for the way we 
f look at things, as long as it is coupled with effectiveness and • ~' : 

irtpact to' measure and monitor the changes in the populations we 
intend to serve. We can also measure changes in the deliv^fy of 
v -""service , and for a service industry , that should be a part of pur ; ^ 




Answer s Harriet Talmadge, in the most recent Encyclopedia of 
■ ■■ Educational Research,, has an article on evaluation of programs 
™^ -^tha^ 

the benef itHeost perspective in evaluation programs , She catego-^ 
rizes the perspectives we bring to valuing, which is, of course, 
what evaluation is about, and identifies the benefit-cost 
perspective as, only one of many. If we were building a model of 
the world from the point of view of an e valuator, we would have a 
set of arrows coming in that are valuing points of view. One of 
y them is the social^political, then there 1 s the personal, the 
long-range, and short-range. The cost-benefit perspective is 
there, and because of some of these methodological struggles, we 
just haven't, been able to bring that valuing point of view to 
bear' on educational decision making very well, yet. But I think 
we can. It's not just by looking to economics and their mathe^ 
matical models and systems, \Log-de linear equations, dynamic 
programming, etc. We -can look beyond that , just with common 
sense, If we could just display costs accurately at the same 
time we < display gains and outcomes we f d be happier and better 
decision Aakers. • • 

■ t Questions As I listened to the presenters, I was .struck by 
the themes that emerged from each one of them. Bob said that we 
jpught to provide information or get information from the building 
level* I liked what Zend said rt We need to evaluate alternatives _ 
before we implement — you can't just implement a program without 
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looking at it. Experiment, ". In rtermi of costs, what I think we 
can begin to do at the building level is to compare costs, do 
"what^ifi 11 analysis, and generate so®© alternatives i With the 
advent of miorooompu ters f it seems to me that some of nthe 
interesting . and fascinating things to do ' |fr directed at ; the 
managers "What if we buy computers and yenlarge class si^e?" and 
"What if we do this? - That 's where we could' do some of the < 
analysis rather than obtaining actual effectiveness measures* ' 
Answer s There * 11 probably be a laboratory created here in 
the c i ty for ;• that kind of thing, if you *re seriously interested 
in pursuing it. We are considering re instituting a program 
called the . instructional Improvement Fund in " the Portland system, 
which will create a pool of- monies accessible to people at the 
bu i Id ing le^ve 1 . To apply for the monies wi 11 be a fa ir ly 
uncomplicated application process : a min i-grant process . There 
is a one-page form to be filled out, which tells what the 
proposed outcomes are, the benefits to students, how to evaluate 
it, what the school is going to do and what it will cost. The 
dec is ioift -mak ing process to evaluate those proposals would 
constitute. a laboratory in which we could look at the process by 
which people in the, buildings decide what was ja reasonable cost 
for an improvement, and at the process we use at the central 
level tof trade off among those things would also be interesting i 
If you can find a focus for tha^research, you have data 
available* 

Answers I'd like to suggest that the other presenters take a 
look at some of the kinds of decision areas districts face, and 
some of the kinds of processes at work in the schools . We've 
heard two or three of them here this morning. I think the 
program review process Beaver ton has is exemplary . I think the 
reduction process Portland has is exemplary. I think that the 
program change objective concept with your program improvement, 
Walt, is exemplary. Those are neat ideas others could use to 
modify for their own use . ■■ . . 

- \',[':a'' : ' : ' v V''-'; ^V' • 



There are some Interesting issues that came out of Colorado 
to help shape priorities in the time of fiscal constraint and 
reduction, - How do you weigh 'different kinds of programs? How do 
you order them? It seems to me that those kinds of issues would 
be very, useful - to both the practitioner -and the researchers. - : 

■ Questions Fftft qf whmt we've ; betn loolcing it is the:;foriDal> T ^ 
comparative, cost-e f f ect iveness study per se>- doing it sort of as 
a set-aside , separate formal activity .. *: What percent of ' the 
problems that you encounter represent a formal study where 
writing a final report would be worth the trouble and time that 
it would take to do it? v Are we talking about half of these 
cases; five percent of these cases , or what? ^ *• t 

■.. Answer : I don't know what you mean by "these cases." 

under which we need to be doing better cost-analysis work, We 
seem to generally agree that there's a lot to be done there. 
Part of what we're looking a't, and part of what the economists 
are talking about is stand-alone, formal, comparative studies of 
program alternatives: v what the evaluators talk about when they } 
talk about randomized field experiments in evaluation-^a full- 
blown approach where you really need to put everything into it. 
One thing our : program is trying to assess is what kin^ of needs 
are out there, and for what kinds of cost work? So", for this 
formal, full-blown, type of study, what percent of the time when 
you have cost problems woulB this kin4'of a study be a reasonable 
alternative to be considered? 

Answer, s toper ience- has indicated to me as we worked our way 
through an intensive review of any program that /there were a 
number of things needed* One was time. We had to narrow down, < 
the scope of programs we would take a look at in those terms. We 
legitimized tills by the observation that we make few fundamental 
changes to begin with. Most of the time we just diddle around, 
and life goes on, to put it in pedestrian terms. So, if we're 
really going to make a substantive impact, why not take the 
time? Why not pour the resources to it, and focus and concen- 
trate on a particular area, which would include the kind of ; 
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^©^^dataya^ talking about* Bteybe? pa -"a.' 

result of ttxat, you'd make little or no change for other ^reasons* 
Nevertheless^ '.. + 1 thinif it is worth .narrowing the scope and 
attention, recognising that you can't solve all of the questions 
in all of the areas? as you might need to. v You may even have 
'■■ sdme -short-term survival things you have to; do, like out aii of 
^the^programs tv^ peMent* You 1 re going to do that "anyway, 
just to stay alive * My point is that % think It useful and 
necessary to take time if we 1 re talking: about good, descriptive 
cost data , comparisons of the development of the tools §. tedh«- \; : 
niques and formats jot displaying it, etc. It can have a 
substantial impact. ~-~ 

One of the interesting th ings I founc^rtsv that people are 
hungry to l>in a diecision^^ 

and our school clos ing deals ions would be an example of this. We 
have seven basic criteria of this, two of which really can be 
dembnstratable in highly qualif'iable terms* One is economics of 
operation, and the other has to do with demographics. After it v s 
all said and done? the decisions are usually legitimized . The 
point is, the/ attention the people give the criteria is probably 
disproportionate to the credit they get in making the overall 
decisions. So we run that danger? that's possibly what Don was 
talking about. Once you get the data display and work with it, 
you can see it and can compare. It 1 s just lilce looking at basic 
achievement data y you -walk .xight in on it and forget about every- 
thing else. Frankly, that 1 s a problem we've run -into in some of 
these cases? where we 1 ve done a pretty decent job of breaking out 
the costs and comparing them. \ 

Question z My question as to the level off odst^analysis . 
technology and if it's worth the effqrt, can be compared to open 
heart surgery* We know tha all surgeons don * t do open heart 
surgery, and the incidence of such surgery in normal surgical, 
practice is not high* I'm sort off asking that kind of question 
about cost-e f f ecti veness analysis. Yes, we all need to do better 
cost work, to use it more, etc., but how many of us really ought 
to be doing the formal stuff, and how many times would we use 



: V ^ it? Qneer ai year*- imam a ^nthl' : What » s it worth to ; l^ar^ ^bbut > 
■ ■^:^aMAio. do cost-^aiyseil* i^ 1 s a tos tlyV ; time J OTnsiiroiiig J act ? 
.-.•f* i Question ; If I heard Walt correctly, the broader and more • 
^encompassing need involves d isplay technique s -for describing 
'costs before you even think about the oost-effeetiveness, -, ? 

•v analysis **'•"■ ild be interested in hearing from Larry how much of , • ■ . 
is training as an economist was necessary for most of thes^ - : ^ - 
y Studies f because most of ^em'are cost description studiesV^^^^-^ " 
' ^ 1 : ; Answer = Most of them are cost-description studies, so tjm ; : 
.; need for special training is limited, i However , the fflore detailed 
: V we got iptb the post description,* the ino*e the special training . 
5 "became 'necessary. The roost detailed cost descr iptions are 
\-5v.V. Studies 1 and la, which are the transportation studies. The V 

.%/-<•: feasibility studies* come in for some severe critic ism because of 
the nature of the cost, which is probably, at best, superficial, 
•/v ; Question s It seems that this is where an accountant ■ s point 

f of view is important- * ... ^ 

f - " : " Answer s To ascertain what the components of toe costs are , s 
V..: yes. There are other more sophisticated issues, though, as Walt 
.V brought up- For example, thje issue of opportunity costs— if we 

. buy a computer , what are we giving up in the way of aides, text- 
f v books, or library resources? What kind of ;a per iod of comparison 
do we adequately use? • 
Question s Nick, I think your question is^ "Is cost- 



effectiveness analysis cost-effective?" Because of some of^the 
thing's, teat were said today, the- real question ought to be under 
what conditions is it* cost-effective? And I heard from each of 
you different ideas about when it might very well be 
cost-effective, zeno ■ m first comment about instructional 
programming and support programs doesn*t explain everything* but 
it was one key element. If you know something about methodolo- 
gies in different areas, you know why it's easier to do 
cost-effectiveness studies within areas of transportation, etc. 
There's not a lot 'of disagreement about that. 

■ ■ ■ . ' '<■; ; ' : AT * . : . ^ : .i . ; 



ERIC 



c : The second obsetyatibh is thatvthersr: Wre a number of ■ :■. 
comments about the rationality of m 
from the 1 view that org 

use :o£ cost-analysis models to the view that 'organizations ape; 
ina Iter ably irrational and loosely coupled. The saving grape of 
laHTnai^ irrationality/ 
wiiicfi; itfians there are situations in which rational models may 
w^k reasonably well. I thin^ helpful guidance can be draw^ but - 
of \what each of you suggested about the conditions under which, 
cost^ffecti^nejis ana3^sis Mght be rooted, / : -:v' 7 v 

; ' I Jiave to take issue with Walt *s comment that the emphasis 
-Should be placed on the development of measures. Improving the 
quality of. outcome measures F improving the qualify of cost , 
estimates, is a very costly^activity itself apdmay not always be 
worth the effort. I agree that measurement development repre^ 
sents a challenge to doing bost-*ef fectiveness analysis, but 
evaluations that require a iarcje investment up front in order to 
even get a study off the grojund may not be worth the money. - Many 
of the things that Larry's involved in, and Don talked, aboilt, 
seemed to be immediately more cost-e£f ective because they were 
based on readily accepted measures of outcome. 

Answer t I f d like to react to the question that Nick asked. 
First of all., it's obvious that there is a tension between - preci- 
sion in an analysis versus the T quick and dirty. t^hope we dow't 
ever lose the quick and dirty* I think we'd be surprised at the 
kind of value for us to be able to have someone go off in a 
corner and reflect about a problem so that the decision maker 
will not be bombarded by data , when facing a - tough decision ,j * 
Cost-effectiveness analysis is one of the instruments in our 
tool chest, I Hope, because it has a lot of visibility and power 
in terms of a major or large-scale decision, we won't forget 
these other ; tools we * ve examined . It seems to me that one of our 
problems is the capacity of our managers— getting " them to use the 
stuff once it is* prepared, bne of the things that came home to 
me in last spring's project was the fact that if- people have 



access tor someone else * s analysis, they * 11 borrow it^ and "adapt it 
for their own' use, which provides both power and efficiency. '? 

We went mrpjand^ the state and ran into district after district 
that had a j copy of the Beaver ton study on buses and adopted it - 
for their own use. We had some very small districts) that V' ; - 
i typically would not jbe expected to conduct ;.th is k indl of study > " 
but they felt reasonably comfortable about the decisions they 1 
wer.e making based upon- the Beaver ton analysis. My concern is. 
Can people use it? v and, Do they know where to get samples? V My 
last point is 'that, generally, local "school systems can*"t handle 
very many analysis at a time ,; because of lack - of staff. There 
must be ways we can w share and stretch our resources* : ' : 

"Answers Nick, you asked for percents of the need f br formal 

cost-reff ectiveness analyses* I would guess that, right now, if ] 

you think of a model of district decision making, for about 5 to 
15 percent of decisions we make , cost comparisons are of para- 
mount concern- These would be situations where we* re looking for 
the* least expensive alternative, given some plan of action that 
comes up to minimum requirements . We have a business agenda at - - 
every boajcd meeting every two weeks that includes a list of 100 
decisions that we made, such as whether to buy this kind of 
aluminum coating for $28,000 versus this kind for $35,000, 
because it met minimunr specs, or what is the least expensive way 
to run those buses* We know we 1 re going to run them* We know 
how many kids we have to carry. We have a policy that tells all 
of that; we just want the cheapest way to do it* 

What I would guess, though, is that if we looked at the 
degree which those decisions were being made, based on careful 
well-designed, well-planned studies of one sort or another, it 
would be a fraction of them. And if we looked at toe percent of 
those in which the R&D unit of the district was an active and 
helpful partner or leader of the process, it would be an even 
smaller fraction. I think this is going to have to expand within 
toe next five years, or else we 1 re going to find fewer than the 
800 R&D units we now have in the 16,000 school districts in the 
country- 



ERIC 



The next thing I'd .* look mt would be cost studies or decisions 
in which cost comparisons are significant* X would guess thf^p^ 
that would take up the next 60 to 85 percent of the decisions "we 
.make for our .board , for a total between 50 and 8 d percent of the . 
decisions. / Those are things wlier# we ! re looking alternative ^ 
education, -or educational support programs; or actions that have 
differing cost duplications and effectiveness levels , such as pur 
GAI study- The primary question isn't how much the CM cost. 
The question is, how can we most effectively implement cbmpu te r s 
and other high te chno logy in the schools, but we want to use cost 
as a *pr i sm on that * : ' •" " t "' 7 * " 

: The last *th ing is that there are another 15 to 35 percent of . 
cases in which cost comparisons are not very significant. ; This 
would be a case when we -re g iyen pre-alloca ted funds , and- we 1 re 
asking which is the most effective program. The funds are 
pre -allocated, so the key issue here is effectiveness and not * 
costs* These are the, decisions Supported more or less and that 
of those I'd say Ran units are probably making a larger 

contr ibution* . .., ■. ■ T - — • , ; '. - . . •• . . -. . - 

Answer t Despite the glory of cost-effectiveness analysis™! 
see little or no need for it at this time* It seems to me that 
if we 're going to develop it, I would like to see the descrip- 
tions of costs and program effects at a building level, and 
that 1 s quite different from cost and effect analysis .; ,1 think we 
have a big specter on the . horizon , however , and this is going to 
come from the federal government in Circular 102, that will be 
followed by federal auditors in the very near future. They're 
going to say we should do. what they call "efficiency audits," and' 
they will do audits of- financial expenditures. But they also 
intend to do program effects audits, and try to relate to them. 
I don't know how many of you have been through that kind of 
horrendous behavior * We did one here, at the Lab. To give you an 
idea of how ludicrous they've become ' (at that time we were 
developing several Alaskan readers) , a questionnaire was sent to 
all of the teachers 'in Washington, Oregon, Idaho, Montana and 
Alaska to see how many had heard about the . Alaskan readers, 



It was reported ai part of the eff fact! venial that not many off the 
teachers had heard off -them* The implementation through the 
auditing world off this type off audit is going" to cause us tremen- 
dous problems unless we are able to have a data base to say "Here 
ar^ the effffeets> and here are the costs" descriptively / iff in no 
other way * Is anyone else afraid about Circular 102? t >H/ = 

Answer: Probably that"; "coupled with the single aud it ^ ^ , 
?obnc#pt> oah bring us right to our knees so we can* t do anything. 

Question: What is a "single audft? ■ < . 

Answer : , They . r coine in and do it all- ; They get ydu all at : ^ ^ 
once "the whole pig, V '\- \ '. . ; - * . 
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